Entity database timestamps

ABSTRACT

In particular embodiments, a system for managing a virtualization environment includes host machines, each of the host machines including a hypervisor, user virtual machines (UVMs) and a virtual machine controller. The virtualization environment also includes virtual disks comprising a plurality of storage devices, and being accessible by the virtual machine controllers. The virtual machine controllers conduct I/O transactions with the virtual disks. The system stores an entity-relationship graph representing elements in the virtualization environment. Each of the elements is represented by an entity-type node in the entity-relationship graph, and relationships between the elements are represented by edges between the nodes.

PRIORITY

This application claims the benefit, under 35 U.S.C. §119(e), of U.S.Provisional Patent Application No. 62/294,980, filed 12 Feb. 2016, whichis incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to virtualization environments.

BACKGROUND

A “virtual machine” or a “VM” refers to a specific software-basedimplementation of a machine in a virtualization environment, in whichthe hardware resources of a real computer (e.g., CPU, memory, etc.) arevirtualized or transformed into the underlying support for the fullyfunctional virtual machine that can run its own operating system andapplications on the underlying physical resources just like a realcomputer.

Virtualization works by inserting a thin layer of software directly onthe computer hardware or on a host operating system. This layer ofsoftware contains a virtual machine monitor or “hypervisor” thatallocates hardware resources dynamically and transparently. Multipleoperating systems run concurrently on a single physical computer andshare hardware resources with each other. By encapsulating an entiremachine, including CPU, memory, operating system, and network devices, avirtual machine is completely compatible with most standard operatingsystems, applications, and device drivers. Most modern implementationsallow several operating systems and applications to safely run at thesame time on a single computer, with each having access to the resourcesit needs when it needs them.

Virtualization allows one to run multiple virtual machines on a singlephysical machine, with each virtual machine sharing the resources ofthat one physical computer across multiple environments. Differentvirtual machines can run different operating systems and multipleapplications on the same physical computer.

One reason for the broad adoption of virtualization in modern businessand computing environments is the resource utilization advantagesprovided by virtual machines. Without virtualization, if a physicalmachine is limited to a single dedicated operating system, then duringperiods of inactivity by the dedicated operating system the physicalmachine is not utilized to perform useful work. This is wasteful andinefficient if there are users on other physical machines which arecurrently waiting for computing resources. To address this problem,virtualization allows multiple VMs to share the underlying physicalresources so that during periods of inactivity by one VM, other VMs cantake advantage of the resource availability to process workloads. Thiscan produce great efficiencies for the utilization of physical computingdevices, and can result in reduced redundancies and better resource costmanagement.

Furthermore, certain virtualization environments may not only utilizethe processing power of the physical computing devices but alsoaggregate storage capacity across the individual physical computingdevices to create a logical storage pool, wherein the data isdistributed across the physical computing devices, yet appears to thevirtual machines to be part of the system that the virtual machine ishosted on. Such systems may utilize metadata, which may be distributedand replicated any number of times across the system, to locate specificdata within the logical storage pool. These systems are commonlyreferred to as clustered systems, wherein the resources of the group arepooled to provide logically combined, but physically separate systems.

SUMMARY OF PARTICULAR EMBODIMENTS

Embodiments of the present invention introduce a user interface todisplay information about entities, their relationships, attributes, andstats in a virtualization environment. A virtualization environment mayinclude a plurality of host machines, each of the host machinescomprising a hypervisor, one or more user virtual machines (UVMs) and avirtual machine controller. The virtualization environment furtherincludes one or more virtual disks comprising a plurality of storagedevices, the one or more virtual disks being accessible by the virtualmachine controllers. The virtual machine controllers conduct I/Otransactions with the one or more virtual disks. A system for managingthe virtualization environment is operable to store anentity-relationship graph representing elements in the virtualizationenvironment. Each of the elements is represented by an entity-type nodein the entity-relationship graph, and relationships between the elementsare represented by edges between the nodes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a clustered virtualization environment according tosome embodiments.

FIG. 1B illustrates data flow within a clustered virtualizationenvironment according to some embodiments.

FIG. 2 illustrates the basic pattern of an entity relationship graphaccording to some embodiments.

FIG. 3A illustrates an example entity-relationship graph for entitiesassociated with a virtualization system according to some embodiments.

FIG. 3B illustrates another example entity-relationship graph forentities associated with a virtualization system according to someembodiments.

FIG. 4 illustrates an example architecture for an entity databaseaccording to some embodiments.

FIG. 5 illustrates an example entity trail for a user virtual machineentity associated with a virtualization environment according to someembodiments.

FIG. 6 illustrates an example context anchor entity ranking according tosome embodiments.

FIG. 7 shows an example client user interface illustrating a list ofvirtual machines in a virtualization environment with associatedattribute values according to some embodiments.

FIG. 8 shows an example client user interface illustrating selection ofvirtual machines in a virtualization environment according to someembodiments.

FIG. 9 shows an example client user interface illustrating filtering ofentities and attributes in a search feature according to someembodiments.

FIG. 10 shows an example client user interface illustrating groupingentities and attributes in a search feature according to someembodiments.

FIG. 11 illustrates a UI displaying labels applied to the selectedVM-type entities.

FIG. 12 shows an example client user interface illustrating a statisticsdisplay based on grouping and derived metrics according to someembodiments.

FIG. 13 illustrates sample statistics for the overall virtualizationenvironment that may be maintained by the entity database according tosome embodiments.

FIG. 14 illustrates a UI displaying a graphical overview representationof current state and other information for the selected and groupedVM-type entities.

FIG. 15 illustrates a UI displaying a graphical overview representationof current state and other information for a filtered subset of theVM-type entities currently existing in the virtualization environment.

FIG. 16 illustrates an example of range-based replication according tosome embodiments.

FIG. 17 illustrates a block diagram of a computing system suitable forimplementing particular embodiments.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1A illustrates a clustered virtualization system 100 according toparticular embodiments. The architecture of FIG. 1A can be implementedfor a distributed platform that contains multiple host machines 101 a-cthat manage multiple tiers of storage. The multiple tiers of storage mayinclude storage that is accessible through network 140, such as, by wayof example and not limitation, cloud storage 126 (e.g., which may beaccessible through the Internet), network-attached storage (NAS) 128(e.g., which may be accessible through a LAN), or a storage area network(SAN). Unlike the prior art, the present embodiment also permits localstorage 122 a-c that is incorporated into or directly attached to thehost machine and/or appliance to be managed as part of storage pool 160.Examples of such local storage include Solid State Drives 125(henceforth “SSDs”), Hard Disk Drives 127 (henceforth “HDDs” or “spindledrives”), optical disk drives, external drives (e.g., a storage deviceconnected to a host machine via a native drive interface or a serialattached SCSI interface), or any other direct-attached storage. Thesestorage devices, both direct-attached and network-accessible,collectively form storage pool 160. Virtual disks (or “vDisks”) may bestructured from the physical storage devices in storage pool 160, asdescribed in more detail below. As used herein, the term vDisk refers tothe storage abstraction that is exposed by a Controller/Service VM (CVM)110 to be used by a user VM 105. In particular embodiments, the vDiskmay be exposed via iSCSI (“internet small computer system interface”) orNFS (“network file system”) and is mounted as a virtual disk on the userVM. In particular embodiments, vDisks may be organized into one or morevolume groups (VGs). In particular embodiments, the virtualizationenvironment may include two or more clusters.

Each host machine 101 a-c may run virtualization software, such asVMWARE ESX(I), MICROSOFT HYPER-V, REDHAT KVM, or NUTANIX AHV. Thevirtualization software includes hypervisor 130 a-c to create, manage,and destroy user VMs 105, as well as managing the interactions betweenthe underlying hardware and user VMs 105. User VMs 105 may run one ormore applications that may operate as “clients” with respect to otherelements within virtualization system 100. Though not depicted in FIG.1A, a hypervisor may connect to network 140. In particular embodiments,a host machine 101 may be a physical hardware computing device; inparticular embodiments, a host machine 101 may be a virtual machine.

CVMs 110 a-c are used to manage storage and input/output (“I/O”)activities according to particular embodiments. These special VMs act asthe storage controller in the currently described architecture. Multiplesuch storage controllers may coordinate within a cluster to form aunified storage controller system. CVMs 110 may run as virtual machineson the various host machines 101, and work together to form adistributed system 110 that manages all the storage resources, includinglocal storage 122, NAS 128, and cloud storage 126. The CVMs may connectto network 140 directly, or via a hypervisor. Since the CVMs runindependent of hypervisors 130 a-c, this means that the current approachcan be used and implemented within any virtual machine architecture,since the CVMs of particular embodiments can be used in conjunction withany hypervisor from any virtualization vendor.

A host machine may be designated as a leader node within a cluster ofhost machines. For example, host machine 101 b, as indicated by theasterisks, may be a leader node. A leader node may have a softwarecomponent designated to perform operations of the leader. For example,CVM 110 b on host machine 101 b may be designated to perform suchoperations. A leader may be responsible for monitoring or handlingrequests from other host machines or software components on other hostmachines throughout the virtualized environment. If a leader fails, anew leader may be designated. In particular embodiments, a managementmodule (e.g., in the form of an agent) may be running on the leadernode.

Each CVM 110 a-c exports one or more block devices or NFS server targetsthat appear as disks to user VMs 105 a-c. These disks are virtual, sincethey are implemented by the software running inside CVMs 110 a-c. Thus,to user VMs 105 a-c, CVMs 110 a-c appear to be exporting a clusteredstorage appliance that contains some disks. All user data (including theoperating system) in the user VMs 105 a-c and reside on these virtualdisks.

Significant performance advantages can be gained by allowing thevirtualization system to access and utilize local storage 122 asdisclosed herein. This is because I/O performance is typically muchfaster when performing access to local storage 122 as compared toperforming access to NAS 128 across a network 140. This fasterperformance for locally attached storage 122 can be increased evenfurther by using certain types of optimized local storage devices, suchas SSDs. Further details regarding methods and mechanisms forimplementing the virtualization environment illustrated in FIG. 1A aredescribed in U.S. Pat. No. 8,601,473, which is hereby incorporated byreference in its entirety.

FIG. 1B illustrates data flow within an example clustered virtualizationsystem 100 according to particular embodiments. As described above, oneor more user VMs and a CVM may run on each host machine 101 along with ahypervisor. As a user VM performs I/O operations (e.g., a read operationor a write operation), the I/O commands of the user VM may be sent tothe hypervisor running on the same host machine as the user VM. Forexample, the hypervisor may present to the virtual machines an emulatedstorage controller, receive an I/O command and facilitate theperformance of the I/O command (e.g., via interfacing with storage thatis the object of the command, or passing the command to a service thatwill perform the I/O command). An emulated storage controller mayfacilitate I/O operations between a user VM and a vDisk. A vDisk maypresent to a user VM as one or more discrete storage drives, but eachvDisk may correspond to any part of one or more drives within storagepool 160. Additionally or alternatively, CVM 110 a-c may present anemulated storage controller either to the hypervisor or to user VMs tofacilitate I/O operations. CVM 110 a-c may be connected to storagewithin storage pool 160. CVM 110 a may have the ability to perform I/Ooperations using local storage 122 a within the same host machine 101 a,by connecting via network 140 to cloud storage 126 or NAS 128, or byconnecting via network 140 to local storage 122 b-c within another hostmachine 101 b-c (e.g., via connecting to another CVM 110 b or 110 c). Inparticular embodiments, any suitable computing system 1700 may be usedto implement a host machine 101.

In particular embodiments, the clustered virtualization environment mayinclude a distributed entity database for storing datacenter entitiesand the relationships between them, as represented by anentity-relationship graph. An entity database may support such anentity-relationship model by storing information related to theentity-relationship graph, such as entities, entity relationships, andtime series information and statistics. The database may be sharded(a.k.a. distributed) across multiple nodes, and may have a query modeloptimized for the entity-relationship model that supports queryingcurrent and historical data, queries for keyword searches, watches fornotifying clients on entity updates, and synchronization of data betweendatabase instances.

FIG. 2 illustrates the basic pattern of an entity-relationship graph200, which may comprise an entity-type node 210 (e.g., a parententity-type node) connected to a child entity-type node 212. Inaddition, a plurality of activity-type nodes may be connected toentity-type node 210, including an action-type node 220, an alert-typenode 222, a metrics-type node 224, and an attributes-type node 226. Asan example and not by way of limitation, each entity-type node (e.g.,entity type node 210, child entity-type node 212, etc.) is connected toat least one of the activity-type nodes. For simplicity, only one nodeis shown in the basic structure of the entity-relationship graph 200,but this disclosure contemplates that each node type can comprise one ormore nodes of that node type.

The schema of an entity-relationship graph (including relationshipsbetween entities) may be defined at runtime, and the entity-relationshipgraph may dynamically evolve as conditions change. As illustrated, eachentity may have its own properties. In particular embodiments, eachentity may also inherit properties from related entities (e.g.,inheriting available actions from a parent). In particular embodiments,certain properties of an entity may represent the aggregated propertiesof its child entities (e.g., metrics data for a cluster entity maycomprise the aggregated metrics data from its child entity nodes).Values for different attributes and/or metrics data for an entity may begenerated by different sources (e.g., different services running in thevirtualization environment and/or other entities). Particular entitiesmay retain historical data relating to changes in its properties and/ormetrics data (e.g., tracked statistical information). Such historicaldata may be retained together with timestamp information so that theevolving state of the virtualization environment is captured in anentity trail, as discussed further in relation to FIG. 5.

FIG. 3A illustrates an example entity-relationship graph 300 forentities associated with virtualization system 100. As shown in FIG. 3A,entity-type nodes may include a multi-cluster-type node 330, acluster-type node 332, a host-type node 334, a virtual-machine-type node336, a virtual-disk-type node 338, a protection-domain-type node 340, acontainer-type node 342, a network-interface-card-type node 344, adisk-type node 346, and a storage-pool-type node 348. In particularembodiments, the direction of the edge connecting the nodes indicates achild-parent relationship (e.g., the arrow goes from a child node to aparent node). As an example and not by way of limitation, cluster-typenode 332 is a child node of multi-cluster-type node 330 and the parentnode of protection-domain-type node 340 and container-type node 342. Asanother example and not by way of limitation, virtual-machine-type node336 is a child node of protection-domain-type node 340, host type node334, and container-type node 342, and a parent node of virtual-disk-typenode 338.

FIG. 3B illustrates another example entity-relationship graph 300 forentities associated with virtualization system 100, together withexample attributes, actions, metrics, alerts, and other informationassociated with the entities. As an example and not by way oflimitation, cluster-type node 332 may be associated with action-typenodes 350 (e.g., settings, etc.), alert-type nodes 360, attribute-typenodes 370 (e.g., cluster name “C1”) and metric-type nodes 380 (e.g., I/Ooperations per second (“IOPS”), latency, etc.). As another example andnot by way of limitation, virtual-machine-type node 336 may beassociated with action-type nodes 350 (e.g., create, poweroff, clone,migrate, etc.), alert-type nodes 360 (e.g., alert types such as cpu,etc.), attribute-type nodes 370 (e.g., num vcpu, IP address, OS type,etc.), and metric-type nodes 380 (e.g., IOPS, latency, CPU utilization,diskspace usage, etc.).

FIG. 4 illustrates an example architecture for the entity database. Thearchitecture may comprise a Cache Layer 430 on top of a Store Layer 420on top of a Persistence Layer 410 (e.g., underlying data store and/orvolume groups in the storage pool). The architecture may support a queryprocessing process 442, put/delete processing process 444, andwatches/alerts process 446. The architecture may also exposefunctionality via an API 450. Persistence Layer 410 may comprisepersistent non-volatile storage of the database information. Cache Layer430 may comprise storage of information that is expected to be retrievedfrequently or in a relatively short time span, or any suitabletransitory storage before moving data to persistent storage.

In particular embodiments, the Store Layer 420 handles requests from andresponses to API layer 450 to store and retrieve data from PersistenceLayer 410. The Store Layer 420 may handle building and updating databaseschemas by example, registering entity types and metrics types. As anexample and not by way of limitation, Store Layer 420 may register aVM-type entity having an IP Address, Number of Virtual CPUs and OS Typeattributes. Store Layer 420 may also handle creation, update anddeletion of entities and attributes. As an example and not by way oflimitation, Store Layer 420 may create a VM-type and populate theattributes described above. Furthermore, Store Layer 420 may updatemetrics associated with entities.

In particular embodiments, entity updates and stats updates are treateddifferently by the database. As an example and not by way of limitation,entity updates may be atomic and write-through, so that a readoperations will obtain the most recent values associated with entities.However, metrics and statistics (e.g., cache usage, CPU usage, diskusage, and data transfer rate) may be lazily updated in order to improvedatabase performance. Thus, a read operation on metrics may not yieldthe most recent data. Each attribute change of an entity is also storedas a metric so that attribute and metric values can be populatedtogether as time series in an entity trail. In particular embodiments,metrics may be aggregated on hourly basis.

In particular embodiments, the database may store current entity stateinformation as well as an entity trail for some or all entities. Inparticular embodiments, an entity trail may include some or allhistorical values for the entity, along with corresponding timestampsfor each value. As an example and not by way of limitation, a Hostentity may have one child VM entity at a time 1000, and then receive anupdate to initiate a second VM entity at time 1002, and then a third VMentity at time 1006. The database may store the first VM entity withassociated timestamps 1000-1006 and the second VM entity with associatedtimestamps 1002-1006 in the entity trail. In this manner, the databasemay be able to track every entity's transition over time. Although thisexample manner of storing timestamps is described for ease ofexplanation, other suitable manners of storing timestamps may be used,as will be explained further. The entity database may store statisticsindexed by certain dimensions to support efficient querying of statsdata; additionally, the statistics can be de-normalized for fast dataaccess. For example, the database may store statistics indexed by entityID over a longer time range, which allows entity database to efficientlyreturn statistics for a particular entity for long periods of time.Entity database data may also be grouped by entity type and metric,which may enable the database to efficiently return statistics for allentities of a particular type (e.g., all VMs) for a shorter time range.

In particular embodiments, the database may provide mechanisms to queryand retrieve a current state of an entity, as well as the state of theentity at a specific point in the past, or the states within a timerange. In particular embodiments, the database may provide mechanisms toquery and retrieve past metrics. In particular embodiments, the databasealso provide future metrics through extrapolation or any suitable methodof predicting future metrics. As an example and not by way oflimitation, assuming a present time of 1000, a user may query thedatabase for the CPU utilization of a Host-type entity from times900-1100. The database may return a set of values and associatedtimestamps where values for timestamps 900-1000 correspond to observedvalues and values from 1000-1100 correspond to an extrapolation based onthe slope of the observed curve. Metrics may be stored as raw values, ormay be stored in buckets by using downsampling.

In particular embodiments, the database may provide a query languageinterface (e.g., SQL) to retrieve current and historical entity,attributes, and metrics data. As an example and not by way oflimitation, the query language interface may support the following typesof queries:

-   -   Point queries: retrieves an attribute value for a time range        (e.g., fetch num_iops from VM1 for t1 to t2).    -   Filter queries: retrieves entities or attributes based on        filtering using expressions on entity attributes and statistics        (e.g., fetch num_iops for VMs with cpu_usage>50).    -   Grouping queries: retrieves entities or attributes grouped by        entity attributes and statistics (e.g., fetch num_iops for all        VMs grouped by cluster).    -   Sorting queries: retrieves entities or attributes sorted by        parameters    -   Derived metrics: retrieves metrics based on expressions and        rollup.

FIG. 5 illustrates an example entity trail for a UVM entity associatedwith the virtualization environment. As shown in FIG. 5, entity trailinformation captures changes (represented by stars) in configurationand/or state information for the UVM entity. For example, at time 1875,the UVM may have begun serving additional applications previously servedby another UVM that failed, until the failed UVM was restarted at time4000 and the applications were moved back to the restarted UVM. Thecorresponding changes to the UVM's Power_State and Memory_Size_Bytesentity attributes (other attributes may include, e.g., replicationfactor, provisioned CPU, Memory, SSD) may also be captured as part ofthe entity trail information for the UVM. In particular embodiments,tools for an administrator of the virtualization environment may enablethe administrator to view the state of the virtualization environment atdifferent points in time/time periods (e.g., actual past performance orpredicted future performance). Such tools may also enable theadministrator to easily roll back the state of the virtualizationenvironment to a past point in time (e.g., in order to “reset” the stateor perform testing). As an example and not by way of limitation, a userinterface for such tools may simply incorporate a slider 510 (e.g.,comprising an element 510A to select a start point of a time period andan element 510B to select an end point of the time period) to navigateforward and backward in the entity trail. As shown in FIG. 5, inparticular embodiments, the user interface may provide the ability toselect a time period and then display entity trail information in agraphical manner.

In particular embodiments, the database may use a data model where thebasic unit is an “entity” identified by entity id. Entities may beatomically updated. Each entity may have a list of key/value pairscalled attributes. For example an entity may have an id “VM#3,” a “name”attribute with “Webserver” as its value, and an “ip_address” attributewith “10.4.75.23” as its value.

In particular embodiments, entities may be organized into disjoint setscalled shards for the purpose of distributing processing betweenmultiple nodes. In particular embodiments, an entity may belong toexactly one shard. Processing of a shard (e.g., for caching, concurrencycontrol, etc.) may be handled by a single node.

Entity modifications may be recorded to the entity trail. When an entityis updated, the previous value of the entity may be recorded into thetrail. Thus, the trail would have the full history of entitymodifications. Creations, updates and deletions may be treated the sameway. In particular embodiments, when an entity is deleted, a “tombstone”record maintained in the DB as historical data. Entity modifications maybe stamped with monotonically increasing wall-clock-based timestamps.The timestamps may be guaranteed to be unique within a shard. Theguarantee may be implemented by maintaining one integer counter pershard that is atomically updated to the new timestamp with a highervalue with every modification. In particular embodiments, the state ofthe virtualization environment at a time in the past may be resurrectedfrom the captured entity trail.

In particular embodiments, the unique monotonically increasingtimestamps may be used to track replicated changes. Data may bereplicated across multiple database instances for various purposes suchas, by way of example and not limitation, to provide global views,disaster recovery, run secondary analytics applications, etc. Inparticular embodiments, the database may track replicated modificationskeeping the “last replicated” timestamp. Modifications may be replicatedin the order of their timestamps, and after each modification isreplicated, the “last replicated” timestamp may be advanced to accountfor replicated modifications.

In particular embodiments, the database may track timestamp ranges ofreplicated modifications, such that when all modifications that havetimestamps between the bottom of the range and the top of the range arereplicated, then the range is marked as replicated. Replicated rangesmay be tracked in a map that contains non-overlapping ranges. Ifadjacent or overlapping ranges are replicated, the ranges in the map maybe merged together, and thus the representation of replication state maybe reduced. Assuming a scenario where all modifications are replicated(for example, due to updates at every time), the representation wouldonly take one extra integer compared with an approach of keeping trackof the “last replicated” timestamp.

Range-based replication state tracking may provide several benefits tothe database. For example, independent failure modes: if onemodification fails to replicate, other modifications still can bereplicated; with “last replicated” timestamp any failure may stallfurther replication. Furthermore, range-based replication may allowmodifications to be sent via multiple channels in any order, withouthaving to serialize communication, while with “last replicated”timestamp ordering may need to be enforced. Moreover, range-basedreplication may provide the ability to replicate latest modificationsbefore historical data. This may be beneficial for bootstrapping newreplicas, since the current value of an entity may be more important foruser scenarios than the entity trail. In particular embodiments, thedatabase may not provide atomicity across entities, and each entityupdate may be applied independently from other entities.

FIG. 6 illustrates the context anchor entity ranking. As an example andnot by way of limitation, the numbers in the bottom right corner of eachbox show the rankings that the searched entities would get if VM#3 isspecified as a context anchor entity.

In particular embodiments, all ranks may be combined into the relevancerank that is used to sort the search results, so that the most relevantresults may be at the beginning of the result list. In particularembodiments, the relevance score calculation algorithm for each entitymay be as follows:

-   -   for each attribute, an individual attribute rank may be a sum of        the match closeness and the relevant attribute rank    -   an entity attribute rank is calculated as the maximum of        individual attribute ranks    -   the relevance score may be calculated as the sum of entity        attribute rank, related entity rank and context anchor entity        rank.

Although the present disclosure describes calculating the relevancescore for entities in a particular manner, the present disclosurecontemplates calculating the relevance score in any suitable manner.

Entities in the database may be linked in a hierarchical manner using“parent” links. An entity may have any number of parents. For example,an entity with identifier “VM#3” could have a parent entity withidentifier “Node#22,” in which case the “VM#3” entity would have aparent link pointing to “Node#22” entity.

Entities of interest may be retrieved using the above described queryfunctionality. In particular embodiments, a caller may pass a searchsubstring and attributes to match. As an example and not by way oflimitation, if the caller issued a search request withattribute_to_look_for=“name” and substring_to_match=“server”, then thecaller may get “VM#3” that has “name” attribute with the “Webserver”value.

Entities of interest may be retrieved using the above described queryfunctionality. In particular embodiments, a caller may pass a searchsubstring and attributes to match. As an example and not by way oflimitation, if the caller issued a search request withattribute_to_look_for=“name” and substring_to_match=“server”, then thecaller may get “VM#3” that has “name” attribute with the “Webserver”value.

A Search may return results in order of relevance (e.g., most relevantfirst). In particular embodiments, the relevance order may be determinedby a relevance score or rank assigned to entities as a result of aranking algorithm. In particular embodiments, the relevance score may becalculated based on several rankings of the following relevance signals:

-   -   match closeness    -   relevant attributes    -   related entities    -   contextual entities

In particular embodiments, match closeness reflects how similar theresulting value is to the provided search substring. As an example andnot by way of limitation, there may be three levels of match closeness:full match (the value is equal to the provided substring), prefix match(the provided substring is the prefix of the value), and any othermatch. As an example and not by way of limitation, in the case of fullmatch the rank is 3, in the case of prefix match the rank is 2, and inthe case of other match the rank is 1.

The caller of the search request may specify relevant attributes as oneof the ways to specify context for search. For example, the caller mightwant entities that have a match in “ip_address” attribute to be rankedhigher in the search results than entities with matches in otherattributes. In particular embodiments, the rank that is assigned when arelevant attribute matches may be specified by the caller. As an exampleand not by way of limitation, the caller may specifyrelevant_attribute={“ip_address”, 3} to signify that a match in“ip_address” attribute would result in assigning a rank of 3.

In particular embodiments, the caller of search may specify relatedentities as one of the ways to specify context for search. As an exampleand not by way of limitation, the caller might want descendants of“node#22” (e.g., VMs, virtual disks, virtual NICs, etc.) to be rankedhigher than other entities. In particular embodiments, the callerspecifies the rank that is assigned when an entity is a descendant of arelevant ancestor. As an example and not by way of limitation, thecaller may specify relevant ancestor={“node#22”, 3} to signifydescendants of node#22 have a rank of 3).

In particular embodiments, the caller of search may provide contextualor “context anchor” entities that are relevant to the search context.The context anchor entities may affect ranking of the search results. Inparticular embodiments, context anchor entities are identified from thecontext under which the search query was generated. As an example andnot by way of limitation, a search context may be that a user is lookingat a screen that shows VM#3 details and starts typing a search query.The interface may then pass VM#3 as a context anchor entity argument inthe search request. In particular embodiments, the interface may provideauto-complete suggestions (where auto-complete uses search) that includeentities that are related to VM#3.

In particular embodiments, a rank calculation may be performed forentities related to “anchor” entities. In particular embodiments, therank of an entity is the path length between the ranked entity and thecontextual entity in the entity relationship graph. In particularembodiments, the depth of traversal may be limited to a predeterminedlength (e.g., 3).

In particular embodiments, the relevance rank calculation algorithm maycombine the above ranks. In particular embodiments, all entitiesinitially have a rank of 0 and a context anchor entity itself may beassigned a predetermined rank (e.g., 3). In particular embodiments, theparents and children of the context anchor entity are assigned a lesserpredetermined rank (e.g., 2) and the parents and children of those aneven lesser predetermined rank (e.g., 1).

In particular embodiments, after an entity's information is updated withnew information, the prior values for the entity are recorded in theentity database, along with associated timestamp that record the timewhen the entity had the prior values. In particular embodiments, theentity database may take a snapshot of all entities periodically (e.g.,every 30 seconds, every 5 minutes, twice a day, etc.). By takingperiodic snapshots, particular embodiments may avoid the burden ofreading logs from the beginning of time in order to get the entire listof available entities and their states at a particular timestamp in thepast. The entity database may also store the entity's statistics, suchas, resource usage (e.g., CPU utilization, memory availability, storageavailability, etc.). In particular embodiments, entity statistics may bedownsampled by aggregating the information (e.g., taking the average,sum, minimum, maximum) over a time period. In this manner, instead ofstoring the prior values, the entity database may record lessinformation where the information is representative of the prior values.In particular embodiments, such downsampling may continually occur asthe entity statistics are collected (e.g., in order to reduce the volumeof such logged information). In addition, older raw and/or downsampledstatistics may be discarded as they expire (e.g., fall out of a rollingwindow of time, such as six months after the present time). Inparticular embodiments, certain common aggregation operations may beprecomputed prior to receiving a relevant query in order to reducelatency.

In particular embodiments, the entity database may model real worldobjects as entities. Each entity may be identified by an entity uniqueidentifier, such as a globally unique identifier (GUID). As an exampleand not by way of limitation, each entity may have an (entity type, id)pair and contain one or more attributes and metrics, each with a nameand values represented as (time, value) pairs. As an example and not byway of limitation, an entity for a VM object may have an id “vm1”, anattribute “vm_name” with a value (t1, “controller vm1”), and metric“cpu_usage_percentage” with a value (t2, “60”). The attributes may alsobe used to express relationship between entities. In the above example,the VM entity may have a “node” attribute with a value “node1”, whichmeans that node1 is the parent entity of vm1. In particular embodiments,attribute updates may be fail-safe whereas metric updates may be lossy.

In particular embodiments, the entity database may provide a mechanismfor transforming different database schemas into a single, unifiedschema. As an example and not by way of limitation, one or more databaseschemas describing portions of the virtualization environment may becombined into a unified schema that contains all information of theenvironment. In particular embodiments, rules describing on how totransform database schemas into a unified schema may specified in atemplate or configuration file and may include the following:

-   -   1. entity id conversion,    -   2. attribute and metric addition/deletion/renaming,    -   3. conversion of attributes to metrics,    -   4. suppression of entity/attribute updates while still        supporting addition of metric information, and    -   5. defining version-specific schema conversion.

Entity ID conversion. In particular embodiments, the entity databaseserver may receive entity data associated with a portion of thevirtualization environment. As an example and not by way of limitation,the entity database server may receive data from multiple registeredclusters of computers that user a different database schemas. Theentities received by the entity database may have cluster-wide (but notglobally) unique IDs. In particular embodiments, before we add thoseentities to the entity database, the entity IDs are converted intoglobally unique identifiers. In particular embodiments, the entitydatabase makes use of an attribute value that is globally unique. In theabove VM example, the entity database may derive the entity ID from a“vm_uuid” attribute. The resultant VM entity may then have a“transformed” globally unique entity ID.

Attribute/Metric addition, deletion and renaming: In particularembodiments, there may attributes or metrics that do not havesignificance in the unified data model, and the entity database mayignore them. As an example and not by way of limitation, an older schemamay have a “node” entity with a “vm_ids” attribute that specifies a listof VMs attached to the node. However, the entity database may includeattributes that define a parent-child relationship, as explained above,and therefore a list of attached VMs may be redundant. As a result, theentity database may ignore the list. Similarly, attributes may berenamed to create the parent-child relationship in the entity database.As an example and not by way of limitation, the older schema databasemay contain a VM entity, vm1, that has a “node_uuid” attribute withvalue “node1.” In the entity database, the corresponding attribute wouldbe “node”, so the entity database may rename this attribute name to“node” before adding it to the database. The attribute value may remainunchanged. Therefore, after the rename operation, the entity databasemay have a “vm” entity that has an implicit reference to a “node” entitywith id “node1”.

Conversion of attributes to metrics: Since the entity database mayprovide fail-safe guarantees for attributes, such updates may be moreexpensive than metric updates. In particular embodiments, the entitydatabase may convert attributes to metrics. As an example and not by wayof limitation, if the entity database receives updates for certainattributes very frequently, those attributes may be converted to metricsin order to reduce the number of fail-safe updates needed. In particularembodiments, metrics updates in the entity database are notwrite-through.

Suppression of entity updates: In particular embodiments, the schematransformation may be used to suppress entity updates while stillallowing metrics data into the entity database. As an example and not byway of limitation, an entity attribute may be transformed into a metricstype attribute when the entity database does not want fail-safe updatesfor the attribute.

Version-specific schema conversion: In particular embodiments, theschema conversion for the entity database may be driven by aversion-specific configuration. As an example and not by way oflimitation, the entity database may be configured to suppress entityupdates when the data is received from a cluster running a specificversion of software (e.g., Nutanix OS). In particular embodiments, thecluster version may be specified in the configuration file as a regularexpression string.

In particular embodiments, a “user interface” (UI) framework may be usedfor “dynamically” presenting entities in a virtualization environment,to permit relevant information about particular entities to be presentedto a particular user or client. In particular embodiments, the frameworkmay enable presentation of any arbitrary entity or resource or group orcombination thereof (homogeneous or heterogeneous). In particularembodiments, the framework may retrieve or construct an entity schemaand an object that describes the entity or entities. Based on theschema, the framework may be able to generate visualizations to presentthe data. In particular embodiments, the framework may be alsodynamically able to filter, group, sort, rank/order, and color-code theentities and/or resources, as well as providing the functionality totake actions on a selected group of items. In particular embodiments,the framework may also provide different types of graphicalrepresentations of entities and/or resources in the virtualizationenvironment. In particular embodiments, the framework may provide an APIand/or libraries enabling creation of customized versions of the GUIfeatures. In particular embodiments, the following components may makeup the framework:

Dynamic Entity Presentation: In particular embodiments, arbitrary entitytypes may be presented into the framework with no hard-coded data forhow to present the entity information. Each entity type may be describedby its schema, and all the other pieces of the framework may adhere tothis schema in determining the presentation method.

Visualizations: In particular embodiments, a framework may be a pluginframework for displaying visualizations of the entity-relationship graphin a user interface so that the full user interface may be created basedon the definition of a few content files. In particular embodiments, theframework may allow a finite number of types of visualizations or userinterfaces, with the option to add more. As an example and not by way oflimitation, the framework may initially allow three types ofvisualizations, but more visualizations may be plugged in. In particularembodiments, each visualization may have a consistent interface and theframework keeps all visualizations synchronized.

Perspectives: In particular embodiments, the framework may provide forgrouping of related attributes of an entity type into perspectives. Inparticular embodiments, the framework may permit an additional dimensionof presenting related information. As an example and not by way oflimitation, instead of a flat presentation of all attributes on a singleuser interface, users may specify groups of attributes of interest to beplaced into different perspectives (e.g. layers) and switch betweenthem.

Filtering: In particular embodiments, the framework may automaticallygenerate and present one-click filtering for the entity type. Inparticular embodiments, the one-click filtering may provide a very fastand smooth presentation of data to the users with minimal user input. Inparticular embodiments, for continuous data, filter ranges may bedynamically created based on the data available and presented to theuser.

Coloring: In particular embodiments, entities and/or resources may becolored by attributes to distinguish the entities and/or resources forpresentation. In particular embodiments, the use of color may provide avisual data exploratory interface. As an example and not by way oflimitation, an interactive legend may be provided to identify each colorgroup to the user and what a particular color signifies.

Grouping: In particular embodiments, entities may be placed into groups.In particular embodiments, determination of a grouping for an entity maybe based on user input. In particular embodiments, multiple users mayprovide input for a particular entity. The UI framework may determinewhether the multiple inputs can be reconciled. If not, the UI frameworkmay select a “winning” user input based on information known about eachuser. As an example and not by way of limitation, two users may provideinputs to define a particular entity, where one user is designated a“premium” user. The UI framework may prioritize the premium user's inputover another user. In particular embodiments, the determine of a groupfor an entity may be automatically determined by the UI framework. As anexample and not by way of limitation, entities may be placed into groupsbased on common activity-type nodes associated with those entities. Inparticular embodiments, one entity may be associated with a plurality ofgroups. In particular embodiments, the visual presentation of a group ofentities may further include information regarding the children of theentities in the group.

Point in time analysis: In particular embodiments, each set of datapresented in the UI framework for searches and presentation of entitiesand/or resources may correlate to a point in time or time period. Inparticular embodiments, the entity browser may show the latest entityinformation by default. In particular embodiments, using UIconfiguration features, the browser may present data at any arbitrarytime range. In particular embodiments, the UI framework may provide theuser with elements to navigate back and forth through different pointsin time through the presentation of entity information.

Selection Set: In particular embodiments, when browsing or searching forentities and/or resources, the user may select one or more entities orresources and use them for later analysis. In particular embodiments,the UI frame work may treat the selected set as a dynamic set ofentities that may adhere to all filters, group-by, coloring, andvisualization features of the entity browser.

Labels: In particular embodiments, when browsing or searching forentities and/or resources, the UI framework may enable the user toselect one or more entities or resources, and further create and applylabels to the selected entities or resources. As an example and not byway of limitation, the user may submit a complex query bringing upsearch results of the query, and then apply a bulk label to some or allof the entities and/or resources in the search results. in this example,the UI framework may subsequently enable provide the ability to display,filter, group, sort, rank/order, color-code, and/or take actions on theentities and/or resources by label. In particular embodiments, more thanone label may be applied to any given entity or resource.

Actions: In particular embodiments, when browsing or searching forentities, the user can select entities and apply actions to the selectedentities and/or resources. In particular embodiments, a “shopping-cart”style feature may enable the user to perform multiple rounds ofselecting entities and/or resources (e.g., after searching, browsing,grouping, filtering, sorting, etc.), and then perform a single actionupon all entities and/or resources in the “shopping-cart.” In particularembodiments, the UI framework may then execute all designated actionsfrom the shopping-cart on the designated entities and/or resources.

FIG. 7 illustrates an example embodiment of the UI framework with adisplay of all VM-type entities currently existing in the virtualizationenvironment. The list of entities may be sorted by any of severalattributes 720, including, for example, name, host machine/hardwarenode, storage (total capacity or available), CPU usage, memory usage,status, or available actions. The list of displayed items (e.g.,attributes, properties, statistical data, etc.) may differ betweendifferent entity types. In the example of FIG. 7, a user may also selectother entity-types 710 to view the entities of those types in thevirtualization environment. In this example, the user may select storagepools, containers, hosts, disks, protection domains, remote sites, NICs,virtual disks, or snapshots.

FIG. 8 illustrates a query results UI framework enabling selection ofcertain VM-type entities currently existing in the virtualizationenvironment. In the example of FIG. 8, a user may click on checkboxes810 next to each entity displayed in order to select the correspondingentity. In particular embodiments, the user may then perform actions orapply labels to the checked entities only.

FIG. 9 illustrates a UI enabling application of various filters to theselected VM-type entities. The UI may include a window 910 to displayvarious filtering options. Filtering options may include filtering bylabel 920, so that a user may select one or more labeled that havealready been applied to the entities, and the UI may present only thoseentities with the selected labels. Another example of a filter may be bycluster 930. The entities presented in the UI may already be assigned toone or more clusters, and the user may select which clusters he or shewould like to view. Another filtering option may be a text search field940 which may enable a user to search for a particular VM by name, if itis known. Another filter 950 may specify a particular attribute of theentity. In the example of FIG. 9, filter 950 may differentiate entitiesbased on their disk capacity, so that a user may filter and view onlythose entities with a certain amount of disk space. In particularembodiments, any other attribute of the entities may be selected, suchas available disk capacity, CPU type or speed, or network bandwidth. Inparticular embodiments, a user may view additional filtering optionsthrough a user element 960.

FIG. 10 illustrates a UI enabling grouping of the selected VM-typeentities by different attributes or properties (e.g., grouping VMs byOS). In particular embodiments, the grouping of VM-type entities may bebased on the application of one or more filters, as discussed in FIG. 9.In the example of FIG. 10, there are two groups 1010 and 1020 presented.Group 1010 includes VM-type entities with the operating system UbuntuServer 14.X LTS, and includes 409 VM-type entities and eight hosts.Group 1020 includes VM-type entities with the operating system MicrosoftWindows Server R2, and includes 155 VM-type entities and five hosts. Inparticular embodiments, a user may further filter within a group, exportthe information for the particular group, or perform any other action ashe or she would for the overall set of entities in the virtualizationenvironment.

FIG. 11 illustrates a UI displaying labels applied to the selectedVM-type entities. In particular embodiments, a corresponding UI mayenable creation of labels that may then be applied to one or moreselected entities. In the example of FIG. 11, the UI may presentinformation about each entity in a separate window. A user may select aparticular entity and select one or more labels to apply to the selectedentity through element 1110 which may appear in response to the userselection of the entity. Once the user has selected one or more labels,the UI framework may apply those labels to the selected entity.

FIG. 12 illustrates a UI displaying group-based statistics and otherinformation for the selected and grouped VM-type entities. In theexample of FIG. 12, the selected groups are similar to the groups ofentities created in the example of FIG. 10. In the FIG. 12 interface,the UI displays attributes and metrics-based information about thecombined set of entities in each group. For example, for the first groupof Ubuntu VM-type entities, the UI displays the total number of hostsand VMs, the status of the VMs (e.g. active, suspended, or off), theamount of storage space being used, the current CPU usage for the groupof entities, and memory usage.

FIG. 13 illustrates a UI displaying overall statistics and otherinformation for all VM-type entities currently existing in thevirtualization environment. In the example of FIG. 13, the total numberof VM-type entities, total disk capacity, total memory capacity,minimum, average, and maximum values for IOPs, bandwidth, and latency,etc. In particular embodiments, a user may select one of the presentedelements and view more information about the characteristic, or aboutthe entity representing that value. As an example and not by way oflimitation, if a user selects the element displaying the maximumbandwidth, the UI may switch the display to present information aboutthe particular VM-type entity corresponding to that bandwidth. Asanother example, if a user selects the element displaying the total diskcapacity, the UI may switch the display to present an individualbreakdown of disk capacity by entity or by group.

FIG. 14 illustrates a UI displaying a graphical overview representationof current state and other information for host (machine) entities. Sucha graphical representation may enable manual selection of displayeditems. In the example of FIG. 14, a UI may display icons (e.g. a circle)for each of the hosts on which one or more VMs have been established,wherein the icons are sorted by power state. Each of the icons mayfurther be colored or otherwise marked to indicate the correspondingentity's current power status, such as on, suspended, or off. Inparticular embodiments, the user may select one or more entities simplyby clicking on the corresponding icon, which may be marked with a checkto indicate the selection. In particular embodiments, once the user hasfinished selecting one or more entities, he or she may performadditional actions or apply labels to the selected entities.

FIG. 15 illustrates a UI displaying a graphical overview representationof current state and other information for a filtered subset of theVM-type entities currently existing in the virtualization environment.This may be similar to the example of FIG. 14 however the VM-typeentities are no longer grouped by an attribute such as host machine. Theicons may be differentiated in appearance to indicate that entity'scurrent status. The user may select one or more entities by clicking onthe corresponding icons, and perform additional actions or apply labelsto the selected entities.

In particular embodiments, in a hyper-converged scale-outinfrastructure, the end user may be faced with the problem of performingone or more operations on a large number of entities that are hosted ina remote/distributed environment. To enable such operations, particularembodiments may provide a workflow or user interface that allows usersto make a single request, which may be internally translated intolocalized requests that may then be performed on each of the individualnodes in the distributed environment.

In particular embodiments, to enable taking actions on different type ofentities, special logic in the frontend may provide pre-validationfeedback before given actions are made available to the user. As anexample and not by way of limitation, a user may want to power off threeVMs of different cluster versions, but one of the three versions thatthe user has selected may not support a power-off action. In particularembodiments, the UI may have special logic to notify the user that suchan action cannot be performed on one of those three selected VMs. Inparticular embodiments, the workflow may enable the user to track theprogress of their action in real time. In order to support a distributedenvironment, particular embodiments may provide resiliency for therequest from the central location by resuming the request from the pointof failure should the central location go down.

In particular embodiments, the workflow may provide a sophisticatedvisualization of the progress of these tasks by automatically convergingthe progress of individual tasks into a parent task for easy perception.As an example and not by way of limitation, a task structure may be asfollows:

{    “status”: “kSucceeded”,    “internal_task”: false,   “cluster_uuid”: “27f1b89f-8b56-458b-a082-fa8bf6c51452”,   “internal_opaque”: “<bytes represent internal objects>”,   “logical_timestamp”: 6,    “deleted”: false,    “subtask_uuid_list”:[“4af3a50f-a353-47bf-92a1-89431d5c5dab”],    “component”:“kPrismCentral”,    “start_time_usecs”: 1455234405731588,   “percentage_complete”: 100,    “canceled”: false,    “sequence_id”:7,    “last_updated_time_usecs”: 1455234408734287,   “create_time_usecs”: 1455234405635435,    “operation_type”:“kPrismFanout”,    “complete_time_usecs”: 1455234408734299,   “message”: “”,    “uuid”: “afbe1946-fcbe-45df-aa24-2b1693af8333” }

In particular embodiments, users may be enabled to perform bulkoperations in a vendor-agnostic, heterogeneous data center without tyingthe user to a particular hypervisor. In particular embodiments, this maysimplify the data center operations with crash resistance and real timeprogress tracking. In particular embodiments, in addition to allowingthe user to perform bulk actions from centralized location, users may beenabled to perform heterogeneous actions in the same action flow. As anexample and not by way of limitation, the end user may make use of theuser interface to both power on and power off thousands of VMs from acentralized location with a single request.

In particular embodiments, the workflow may use only one process threadthroughout the process regardless the number of the target entities.Since most of the local jobs are expected done within 1 minute, theprocess is expected to complete within a short duration. In particularembodiments, the workflow may provide a five-minute maximum for all jobsto be done and synced back to the centralized location. In particularembodiments, if some jobs are not completed within five minutes, failureor error may be presumed, and so the process may be terminated in thecentralized location, and the parent task may be marked as “failed” witha detailed reason attached.

FIG. 16 illustrates an example of range-based replication according tosome embodiments. Suppose initially the replica contained no changes,and then a change with the range [990 . . . 1000) is replicated. Thereplica may apply the change and account for it in the replicationstate. As shown by 1610, the replication state could be described by themap of the following ranges {990, 1000}.

Suppose now a change with the range [1100 . . . 1110) is replicated. Thereplica can apply the change and account for it in the replicationstate. As shown by 1620, the replication state could be described by themap of the following ranges {990, 1000}, {1100, 1110}.

Suppose now a change with the range [1000, 1100) got replicated. As thischange is adjacent to the existing ranges, all ranges can now be mergedinto one range. As shown by 1630, the replication state could bedescribed by the map of the following ranges {990, 1110}.

Thus, the range based replication state can represent replicated changesefficiently, and may be more precise than a “latest” timestamp value.

Entities in the database may be linked in a hierarchical manner using“parent” links. An entity may have any number of parents. For example,an entity with identifier “VM#3” could have a parent entity withidentifier “Node#22,” in which case the “VM#3” entity would have aparent link pointing to “Node#22” entity.

FIG. 17 is a block diagram of an illustrative computing system 1700suitable for implementing particular embodiments. In particularembodiments, one or more computer systems 1700 may perform one or moresteps of one or more methods described or illustrated herein. Inparticular embodiments, one or more computer systems 1700 may providefunctionality described or illustrated herein. In particularembodiments, software running on one or more computer systems 1700 mayperform one or more steps of one or more methods described orillustrated herein or provide functionality described or illustratedherein. Particular embodiments include one or more portions of one ormore computer systems 1700. Herein, reference to a computer system mayencompass a computing device, and vice versa, where appropriate.Moreover, reference to a computer system may encompass one or morecomputer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems1700. This disclosure contemplates computer system 1700 taking anysuitable physical form. As example and not by way of limitation,computer system 1700 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a mainframe, a mesh of computer systems, aserver, a laptop or notebook computer system, a tablet computer system,or a combination of two or more of these. Where appropriate, computersystem 1700 may include one or more computer systems 1700; be unitary ordistributed; span multiple locations; span multiple machines; spanmultiple data centers; or reside in a cloud, which may include one ormore cloud components in one or more networks. Where appropriate, one ormore computer systems 1700 may perform without substantial spatial ortemporal limitation one or more steps of one or more methods describedor illustrated herein. As an example and not by way of limitation, oneor more computer systems 1700 may perform in real time or in batch modeone or more steps of one or more methods described or illustratedherein. One or more computer systems 1700 may perform at different timesor at different locations one or more steps of one or more methodsdescribed or illustrated herein, where appropriate.

Computer system 1700 may include a bus 1706 (e.g., an address bus and adata bus) or other communication mechanism for communicatinginformation, which interconnects subsystems and devices, such asprocessor 1707, system memory 1708 (e.g., RAM), static storage device1709 (e.g., ROM), disk drive 1710 (e.g., magnetic or optical),communication interface 1714 (e.g., modem, Ethernet card, a networkinterface controller (NIC) or network adapter for communicating with anEthernet or other wire-based network, a wireless NIC (WNIC) or wirelessadapter for communicating with a wireless network, such as a WI-FInetwork), display 1711 (e.g., CRT, LCD, LED), input device 1712 (e.g.,keyboard, keypad, mouse, microphone). In particular embodiments,computer system 1700 may include one or more of any such components.

According to particular embodiments, computer system 1700 performsspecific operations by processor 1707 executing one or more sequences ofone or more instructions contained in system memory 1708. Suchinstructions may be read into system memory 1708 from another computerreadable/usable medium, such as static storage device 1709 or disk drive1710. In alternative embodiments, hard-wired circuitry may be used inplace of or in combination with software instructions to implement theinvention. Thus, embodiments are not limited to any specific combinationof hardware circuitry and/or software. In one embodiment, the term“logic” shall mean any combination of software or hardware that is usedto implement all or part of the invention.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 1707 for execution. Such a medium may take many forms,including but not limited to, nonvolatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as disk drive 1710. Volatile media includes dynamic memory, such assystem memory 1708.

Common forms of computer readable media include, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

In particular embodiments, execution of the sequences of instructions topractice the invention is performed by a single computer system 1700.According to other embodiments, two or more computer systems 1700coupled by communication link 1715 (e.g., LAN, PTSN, or wirelessnetwork) may perform the sequence of instructions required to practicethe invention in coordination with one another.

Computer system 1700 may transmit and receive messages, data, andinstructions, including program, i.e., application code, throughcommunication link 1715 and communication interface 1714. Receivedprogram code may be executed by processor 1707 as it is received, and/orstored in disk drive 1710, or other non-volatile storage for laterexecution. A database 1732 in a storage medium 1731 may be used to storedata accessible by the system 1700 by way of data interface 1733.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative.

1. A system for managing a virtualization environment, comprising: aplurality of host machines, wherein each of the host machines comprisesa hypervisor, one or more user virtual machines (UVMs) and a virtualmachine controller; one or more virtual disks comprising a plurality ofstorage devices, the one or more virtual disks being accessible by thevirtual machine controllers, wherein the virtual machine controllersconduct I/O transactions with the one or more virtual disks; and whereinthe system is further operable to: store an entity-relationship graphrepresenting elements in the virtualization environment, wherein each ofthe elements is represented by an entity-type node in theentity-relationship graph, and wherein relationships between theelements are represented by edges between the nodes; store an entitytrail comprising prior information associated with one or moreentity-type nodes and associated timestamps; receive, from a clientdevice, additional information associated with one or more entity-typenodes and associated additional timestamps; and record the additionalinformation of the one or more entity-type nodes and correspondingadditional timestamps in the entity trail.
 2. The system of claim 1,wherein the system is further operable to: periodically analyze theinformation in the entity trail to determine resource usage statisticsover time of one or more elements associated with the one or moreentity-type nodes.
 3. The system of claim 2, wherein the statistics arerelated to a CPU utilization, a memory availability, or a storageavailability.
 4. The system of claim 1, wherein the system is furtheroperable to: aggregate information in the entity trail by deriving avalue from information corresponding to a range of timestamps andstoring the value in place of the aggregated information.
 5. The systemof claim 1, wherein one or more of the additional timestamps comprises atime in the future from the perspective of the system.
 6. The system ofclaim 4, wherein the aggregating comprises computing an average, a sum,a minimum or a maximum of the information in the entity trail over therange of timestamps.
 7. The system of claim 4, wherein the aggregatingoccurs continually as entity-type nodes are updated with newinformation.
 8. The system of claim 4, wherein the system is furtheroperable to: discard information in the entity trail after apre-determined time period.
 9. A computer-implemented method formanaging a virtualization environment, comprising: storing anentity-relationship graph representing elements in the virtualizationenvironment, wherein each of the elements is represented by anentity-type node in the entity-relationship graph, and whereinrelationships between the elements are represented by edges between thenodes; storing an entity trail comprising prior information associatedwith one or more entity-type nodes and associated timestamps; receiving,from a client device, additional information associated with one or moreentity-type nodes and associated additional timestamps; and recordingthe additional information of the one or more entity-type nodes andcorresponding additional timestamps in the entity trail, wherein thevirtualization environment comprises: a plurality of host machines,wherein each of the host machines comprises a hypervisor, one or moreuser virtual machines (UVMs) and a virtual machine controller; one ormore virtual disks comprising a plurality of storage devices, the one ormore virtual disks being accessible by the virtual machine controllers,wherein the virtual machine controllers conduct I/O transactions withthe one or more virtual disks.
 10. The method of claim 9, furthercomprising: periodically analyzing the information in the entity trailto determine resource usage statistics over time of one or more elementsassociated with the one or more entity-type nodes.
 11. The method ofclaim 10, wherein the statistics are related to a CPU utilization, amemory availability, or a storage availability.
 12. The method of claim9, further comprising: aggregating information in the entity trail byderiving a value from information corresponding to a range of timestampsand storing the value in place of the aggregated information.
 13. Themethod of claim 12, wherein one or more of the additional timestampscomprises a time in the future from the perspective of the system. 14.The method of claim 12, wherein the aggregating comprises computing anaverage, a sum, a minimum or a maximum of the information in the entitytrail over the range of timestamps.
 15. The method of claim 12, whereinthe aggregating occurs continually as entity-type nodes are updated withnew information.
 16. The method of claim 12, further comprising:discarding information in the entity trail after a pre-determined timeperiod.
 17. One or more computer-readable non-transitory storage mediaembodying software for managing a virtualization environment, thesoftware being operable when executed to: store an entity-relationshipgraph representing elements in the virtualization environment, whereineach of the elements is represented by an entity-type node in theentity-relationship graph, and wherein relationships between theelements are represented by edges between the nodes; store an entitytrail comprising prior information associated with one or moreentity-type nodes and associated timestamps; receive, from a clientdevice, additional information associated with one or more entity-typenodes and associated additional timestamps; and record the additionalinformation of the one or more entity-type nodes and correspondingadditional timestamps in the entity trail, wherein the virtualizationenvironment comprises: a plurality of host machines, wherein each of thehost machines comprises a hypervisor, one or more user virtual machines(UVMs) and a virtual machine controller; one or more virtual diskscomprising a plurality of storage devices, the one or more virtual disksbeing accessible by the virtual machine controllers, wherein the virtualmachine controllers conduct I/O transactions with the one or morevirtual disks.
 18. The storage media of claim 17, wherein the softwareis further operable to: periodically analyze the information in theentity trail to determine resource usage statistics over time of one ormore elements associated with the one or more entity-type nodes.
 19. Thestorage media of claim 18, wherein the statistics are related to a CPUutilization, a memory availability, or a storage availability.
 20. Thestorage media of claim 17, wherein the software is further operable to:aggregate information in the entity trail by deriving a value frominformation corresponding to a range of timestamps and storing the valuein place of the aggregated information.